{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true,
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "# 5.Ajax 数据爬取\n",
    "\n",
    "有时候我们在用 requests 抓取页面的时候，\n",
    "得到的结果可能和在浏览器中看到的不一样：\n",
    "在浏览器中可以看到正常显示的页面数据，但是使用requests得到的结果并没有。\n",
    "\n",
    "这是因为requests获取的都是原始的HTML文档,\n",
    "而浏览器中的页面则是经过JavaScript处理数据后生成的结果，\n",
    "这些数据的来源有多种，可能是通过Ajax加载的，\n",
    "可能是包含在HTML文档中的，\n",
    "也可能是经过JavaScript和特定算法计算后生成的。\n",
    "\n",
    "对于第一种情况，数据加载是一种异步加载方式，原始的页面最初不会包含某些数据，\n",
    "原始页面加载完后，会再向服务器请求某个接口获取数据，\n",
    "然后数据才被处理从而呈现到网页上，这其实就是发送了一个Ajax请求。\n",
    "\n",
    "照Web发展的趋势来看，这种形式的页面越来越多。 网页的原始 HTML 文档不会包含任何数据，\n",
    "数据都是通过Ajax统一加载后再呈现出来的，\n",
    "这样在Web开发上可以做到前后端分离，而且降低服务器直接渲染页面带来的压力。\n",
    "\n",
    " 所以如果遇到这样的页面，直接利用requests等库来抓取原始页面，\n",
    " 是无法获取到有效数据的，\n",
    " 这时需要分析网页后台向接口发送的Ajax请求，\n",
    " 如果可以用**requests**库来模拟**Ajax**请求，那么就可以成功抓取了。\n",
    "\n",
    "**Ajax，全称为 Asynchronous JavaScript and XML**，即异步的JavaScript和XML。 它不是一门编程语言，\n",
    "而是利用 JavaScript 在保证页面不被刷新、\n",
    "页面链接不改变的情况下与服务器交换数据并更新部分网页的技术。\n",
    "\n",
    "有了**Ajax**，便可以在页面不被全部刷新的情况下更新其内容。\n",
    "在这个过程中，\n",
    "页面实际上是在后台与服务器进行了数据交互，\n",
    "获取到数据之后，再利用**JavaScript**改变网页，这样网页内容就会更新了。\n",
    "\n",
    "示例：[http://www.w3school.com.cn/ajax/ajax_xmlhttprequest_send.asp](http://www.w3school.com.cn/ajax/ajax_xmlhttprequest_send.asp)\n",
    "\n",
    "[https://zyk.mingrisoft.com/Develop/view/id/1838/type/6/cid/45.html](https://zyk.mingrisoft.com/Develop/view/id/1838/type/6/cid/45.html)\n",
    "\n",
    "Ajax加载在微博、QQ空间等比较常见\n",
    "\n",
    "测试：打开新浪微博，进入到开发者选项，切换到Network选项卡，随后重新刷新页面，可以发现这里出现了非常多的条目\n",
    "\n",
    "Ajax有其特殊的请求类型，它叫作xhr\n",
    "\n",
    "如果显示的Request Headers中有一个信息为 X-Requested-With:XMLHttpRequest，这就标记了此请求是Ajax\n",
    "\n",
    "图片显示![](AjaxExample.jpg)\n",
    "\n",
    "的微博页面的真实数据并不是最原始的页面返回的，\n",
    "而是后来执行JavaScript后再次向后台发送了Ajax请求，\n",
    "浏览器拿到数据后再进一步渲染出来的。\n",
    "\n",
    "分析出 Ajax请求的一些详细信息了，\n",
    "接下来只需要用程序模拟这些**Ajax**请求，\n",
    "就可以轻松提取我们所需要的信息了。\n",
    "\n",
    "Ajax实际执行的代码："
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    "var xmlhttp:\n",
    "if(window.XMLHttpRequest){\n",
    "    //code for IE7+,Firefox,Chrome,Opera,Safari\n",
    "    xmlhttp = new XMLHttpRequest();\n",
    "}\n",
    "else{//code for IE6,IE5\n",
    "    xmlhttp = new ActiveObject(\"Microsoft.XMLHTTP\")\n",
    "}\n",
    "xmlhttp.onreadystatechange = function(){\n",
    "    if(xmlhttp.readyState == 4 && xmlhttp.status == 200){\n",
    "        document.getElementById(\"myDiv\").innerHTML=xmlhttp.responseText;\n",
    "    }\n",
    "}\n",
    "xmlhttp.open(\"POST\",\"/ajax/\",true)\n",
    "xmlhttp.send()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "Ajax 信息提取基本流程：\n",
    "# 1. 分析请求\n",
    "\n",
    "打开Ajax的XHR过滤器，然后一直滑动页面以加载新的微博内容。\n",
    "可以看到，会不断有Ajax请求发出。\n",
    "\n",
    "效果图：\n",
    "![](Ajaxget.jpg)\n",
    "\n",
    "# 2. 分析响应\n",
    "\n",
    "观察这个请求的响应内容\n",
    "\n",
    "# 3. 实战：分析Ajax爬取今日头条的风景美图并下载到本地！\n",
    "\n",
    "爬取库：**requests**\n",
    "流程：\n",
    "\n",
    "## 1.爬取分析\n",
    "\n",
    "今日头条首页：[http://www.toutiao.com/](http://www.toutiao.com/)\n",
    "\n",
    "查看对应的请求与响应状态\n",
    "\n",
    "我们的目的是要抓取其中的美图，\n",
    "这里一组图就对应前面 data 字段中的一条数据。\n",
    "每条数据还有一个**image_detail(image_list)**字段，它是列表形式，\n",
    "这其中就包含了组图的所有图片列表\n",
    "\n",
    "例子：![](toutiao.jpg)\n",
    "\n",
    "因此我们只需提取Url并下载下来就好,一组图都建立文件夹，\n",
    "文件夹的名称就为组图的标题。\n",
    "\n",
    "接下来，就可以直接用Python来模拟这个Ajax请求，\n",
    "然后提取出相关美图链接井下载。\n",
    "但是在这之前，我们还需要分析一下**URL**的规律。\n",
    "\n",
    "切换回Headers选项卡，观察一下它的请求URL和Headers信息\n",
    "\n",
    "可以看到 ，这是一个GET请求，请求URL的参数有 offset、format等\n",
    "![](query.jpg)\n",
    "\n",
    "这里观察一下后续链接的参数，发现变化的参数只有 offset ，\n",
    "其他参数都没有变化，而且第二次请求的offset值为20,\n",
    "第三次为40，第四次为60，所以可以发现规律，\n",
    "这个offset值就是偏移量，\n",
    "进而可以推断出count参数就是一次性获取的数据条数。\n",
    "因此，我们可以用offset参数来控制数据分页。\n",
    "这样一来，我们就可以通过接口批量获取数据了\n",
    "\n",
    "刷新一次，offset偏移值增加20"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'count': 0, 'return_count': 0, 'query_id': '6537509081622516995', 'has_more': 0, 'request_id': '202007252250020100140170211F25DFDD', 'search_id': '202007252250020100140170211F25DFDD', 'cur_ts': 1595688602, 'offset': 10, 'message': 'success', 'pd': 'synthesis', 'show_tabs': 1, 'keyword': '风景', 'city': '舟山', 'log_pb': {'impr_id': '202007252250020100140170211F25DFDD', 'is_incognito': 0}, 'data': None, 'data_head': [{'challenge_code': 0, 'cell_type': 71, 'keyword': '风景', 'url': 'sslocal://search?keyword=%E9%A3%8E%E6%99%AF&from=&source=search_tab'}], 'ab_fields': None, 'latency': 0, 'search_type': 2, 'tab_rank': None, 'temp_type': 0, 'tab_list': None, 'stability_error': 0, 'synthesis_stability_error': 0, 'gray_page_switch': False}\n",
      "\n",
      "{'count': 0, 'return_count': 0, 'query_id': '6537509081622516995', 'has_more': 0, 'request_id': '202007252250010100140500121650368E', 'search_id': '202007252250010100140500121650368E', 'cur_ts': 1595688602, 'offset': 12, 'message': 'success', 'pd': 'synthesis', 'show_tabs': 1, 'keyword': '风景', 'city': '舟山', 'log_pb': {'impr_id': '202007252250010100140500121650368E', 'is_incognito': 0}, 'data': None, 'data_head': [{'challenge_code': 0, 'cell_type': 71, 'keyword': '风景', 'url': 'sslocal://search?keyword=%E9%A3%8E%E6%99%AF&from=&source=search_tab'}], 'ab_fields': None, 'latency': 0, 'search_type': 2, 'tab_rank': None, 'temp_type': 0, 'tab_list': None, 'stability_error': 0, 'synthesis_stability_error': 0, 'gray_page_switch': False}\n",
      "\n",
      "{'count': 0, 'return_count': 0, 'query_id': '6537509081622516995', 'has_more': 0, 'request_id': '20200725225002010011060088140C6966', 'search_id': '20200725225002010011060088140C6966', 'cur_ts': 1595688602, 'offset': 14, 'message': 'success', 'pd': 'synthesis', 'show_tabs': 1, 'keyword': '风景', 'city': '舟山', 'log_pb': {'impr_id': '20200725225002010011060088140C6966', 'is_incognito': 0}, 'data': None, 'data_head': [{'challenge_code': 0, 'cell_type': 71, 'keyword': '风景', 'url': 'sslocal://search?keyword=%E9%A3%8E%E6%99%AF&from=&source=search_tab'}], 'ab_fields': None, 'latency': 0, 'search_type': 2, 'tab_rank': None, 'temp_type': 0, 'tab_list': None, 'stability_error': 0, 'synthesis_stability_error': 0, 'gray_page_switch': False}\n",
      "\n",
      "{'count': 0, 'return_count': 0, 'query_id': '6537509081622516995', 'has_more': 0, 'request_id': '2020072522500301001504415120128179', 'search_id': '2020072522500301001504415120128179', 'cur_ts': 1595688603, 'offset': 16, 'message': 'success', 'pd': 'synthesis', 'show_tabs': 1, 'keyword': '风景', 'city': '舟山', 'log_pb': {'impr_id': '2020072522500301001504415120128179', 'is_incognito': 0}, 'data': None, 'data_head': [{'challenge_code': 0, 'cell_type': 71, 'keyword': '风景', 'url': 'sslocal://search?keyword=%E9%A3%8E%E6%99%AF&from=&source=search_tab'}], 'ab_fields': None, 'latency': 0, 'search_type': 2, 'tab_rank': None, 'temp_type': 0, 'tab_list': None, 'stability_error': 0, 'synthesis_stability_error': 0, 'gray_page_switch': False}\n",
      "\n",
      "{'count': 0, 'return_count': 0, 'query_id': '6537509081622516995', 'has_more': 0, 'request_id': '202007252250030100140612081624E92B', 'search_id': '202007252250030100140612081624E92B', 'cur_ts': 1595688603, 'offset': 18, 'message': 'success', 'pd': 'synthesis', 'show_tabs': 1, 'keyword': '风景', 'city': '舟山', 'log_pb': {'impr_id': '202007252250030100140612081624E92B', 'is_incognito': 0}, 'data': None, 'data_head': [{'challenge_code': 0, 'cell_type': 71, 'keyword': '风景', 'url': 'sslocal://search?keyword=%E9%A3%8E%E6%99%AF&from=&source=search_tab'}], 'ab_fields': None, 'latency': 0, 'search_type': 2, 'tab_rank': None, 'temp_type': 0, 'tab_list': None, 'stability_error': 0, 'synthesis_stability_error': 0, 'gray_page_switch': False}\n",
      "\n",
      "{'count': 0, 'return_count': 0, 'query_id': '6537509081622516995', 'has_more': 0, 'request_id': '20200725225003010015044033010F43C5', 'search_id': '20200725225003010015044033010F43C5', 'cur_ts': 1595688603, 'offset': 20, 'message': 'success', 'pd': 'synthesis', 'show_tabs': 1, 'keyword': '风景', 'city': '舟山', 'log_pb': {'impr_id': '20200725225003010015044033010F43C5', 'is_incognito': 0}, 'data': None, 'data_head': [{'challenge_code': 0, 'cell_type': 71, 'keyword': '风景', 'url': 'sslocal://search?keyword=%E9%A3%8E%E6%99%AF&from=&source=search_tab'}], 'ab_fields': None, 'latency': 0, 'search_type': 2, 'tab_rank': None, 'temp_type': 0, 'tab_list': None, 'stability_error': 0, 'synthesis_stability_error': 0, 'gray_page_switch': False}\n",
      "\n",
      "{'count': 0, 'return_count': 0, 'query_id': '6537509081622516995', 'has_more': 0, 'request_id': '20200725225003010012061073092956EB', 'search_id': '20200725225003010012061073092956EB', 'cur_ts': 1595688603, 'offset': 22, 'message': 'success', 'pd': 'synthesis', 'show_tabs': 1, 'keyword': '风景', 'city': '舟山', 'log_pb': {'impr_id': '20200725225003010012061073092956EB', 'is_incognito': 0}, 'data': None, 'data_head': [{'challenge_code': 0, 'cell_type': 71, 'keyword': '风景', 'url': 'sslocal://search?keyword=%E9%A3%8E%E6%99%AF&from=&source=search_tab'}], 'ab_fields': None, 'latency': 0, 'search_type': 2, 'tab_rank': None, 'temp_type': 0, 'tab_list': None, 'stability_error': 0, 'synthesis_stability_error': 0, 'gray_page_switch': False}\n",
      "\n",
      "{'count': 0, 'return_count': 0, 'query_id': '6537509081622516995', 'has_more': 0, 'request_id': '2020072522500401019409901240DBF059', 'search_id': '2020072522500401019409901240DBF059', 'cur_ts': 1595688604, 'offset': 24, 'message': 'success', 'pd': 'synthesis', 'show_tabs': 1, 'keyword': '风景', 'city': '舟山', 'log_pb': {'impr_id': '2020072522500401019409901240DBF059', 'is_incognito': 0}, 'data': None, 'data_head': [{'challenge_code': 0, 'cell_type': 71, 'keyword': '风景', 'url': 'sslocal://search?keyword=%E9%A3%8E%E6%99%AF&from=&source=search_tab'}], 'ab_fields': None, 'latency': 0, 'search_type': 2, 'tab_rank': None, 'temp_type': 0, 'tab_list': None, 'stability_error': 0, 'synthesis_stability_error': 0, 'gray_page_switch': False}\n",
      "\n",
      "{'count': 0, 'return_count': 0, 'query_id': '6537509081622516995', 'has_more': 0, 'request_id': '202007252250040101940982322AD8C0A6', 'search_id': '202007252250040101940982322AD8C0A6', 'cur_ts': 1595688604, 'offset': 26, 'message': 'success', 'pd': 'synthesis', 'show_tabs': 1, 'keyword': '风景', 'city': '舟山', 'log_pb': {'impr_id': '202007252250040101940982322AD8C0A6', 'is_incognito': 0}, 'data': None, 'data_head': [{'challenge_code': 0, 'cell_type': 71, 'keyword': '风景', 'url': 'sslocal://search?keyword=%E9%A3%8E%E6%99%AF&from=&source=search_tab'}], 'ab_fields': None, 'latency': 0, 'search_type': 2, 'tab_rank': None, 'temp_type': 0, 'tab_list': None, 'stability_error': 0, 'synthesis_stability_error': 0, 'gray_page_switch': False}\n",
      "\n",
      "{'count': 0, 'return_count': 0, 'query_id': '6537509081622516995', 'has_more': 0, 'request_id': '202007252250040100140292220A20003F', 'search_id': '202007252250040100140292220A20003F', 'cur_ts': 1595688604, 'offset': 28, 'message': 'success', 'pd': 'synthesis', 'show_tabs': 1, 'keyword': '风景', 'city': '舟山', 'log_pb': {'impr_id': '202007252250040100140292220A20003F', 'is_incognito': 0}, 'data': None, 'data_head': [{'challenge_code': 0, 'cell_type': 71, 'keyword': '风景', 'url': 'sslocal://search?keyword=%E9%A3%8E%E6%99%AF&from=&source=search_tab'}], 'ab_fields': None, 'latency': 0, 'search_type': 2, 'tab_rank': None, 'temp_type': 0, 'tab_list': None, 'stability_error': 0, 'synthesis_stability_error': 0, 'gray_page_switch': False}\n",
      "\n"
     ]
    }
   ],
   "source": [
    "from urllib.parse import urlencode\n",
    "import requests\n",
    "def get_page(offset):\n",
    "    params = {\n",
    "        'offset':offset,\n",
    "        'format':'json',\n",
    "        'keyword':'风景',\n",
    "        'autoload':'true',\n",
    "        'count':'20',\n",
    "        'en_qc':'1',\n",
    "        'cur_tab':'1'\n",
    "    }\n",
    "    url = 'https://www.toutiao.com/api/search/content?'+urlencode(params)\n",
    "    try:\n",
    "        headers = {'User-Agent':'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.106 Safari/537.36'}\n",
    "        response = requests.get(url,headers=headers)\n",
    "        if response.status_code == 200:\n",
    "            return response.json()\n",
    "    except requests.ConnectionError:\n",
    "        return None\n",
    "for i in range(10):\n",
    "    print(get_page(str(i*2)))\n",
    "    print()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "这里我们用**urlencode()**方法构造请求的GET参数，\n",
    "然后用requests请求这个链接，如果返回状态码为200，\n",
    "则调用response的json()方法将结果转为**JSON**格式，然后返回。\n",
    "\n",
    "## 接下来，再实现一个解析方法：\n",
    "\n",
    "提取每条数据的image_detail(image_list) 字段中的每一张图片链接，\n",
    "将图片链接和图片所属的标题一并返回，此时可以构造一个生成器。\n",
    "\n",
    "接下来，实现一个保存图片的方法 save_image()，\n",
    "其中 item 就是前面 get_images()方法返回的一个字典。\n",
    "在该方法中，首先根据 item 的 title 来创建文件夹，\n",
    "然后请求这个图片链接，获取图片的二进制数据，\n",
    "以二进制的形式写入文件。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "outputs": [],
   "source": [
    "from urllib.parse import urlencode\n",
    "import requests\n",
    "import os\n",
    "originalocation = os.getcwd()#先确定当前路径\n",
    "# print(\"local rount: \"+originalocation)\n",
    "count = 0\n",
    "def get_page(offset):\n",
    "    params = {\n",
    "        'offset':offset,\n",
    "        'format':'json',\n",
    "        'keyword':'风景',\n",
    "        'autoload':'true',\n",
    "        'count':'20',\n",
    "        'cur_tab':'1',\n",
    "    }\n",
    "    url = 'https://www.toutiao.com/api/search/content?'+urlencode(params)\n",
    "    try:\n",
    "        headers = {'User-Agent':'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.106 Safari/537.36'}\n",
    "        response = requests.get(url,headers=headers)\n",
    "        if response.status_code == 200:\n",
    "            return response.json()\n",
    "    except requests.ConnectionError:\n",
    "        return None\n",
    "# for i in range(10):\n",
    "#     print(get_page(str(i*2)))\n",
    "#     print()\n",
    "\n",
    "\n",
    "def get_image(json):\n",
    "    print(json.get('data'))\n",
    "    if json.get('data'):\n",
    "        for item in json.get(\"data\"):\n",
    "            title = item.get('title')\n",
    "            print(title)\n",
    "            images = item.get('image_list')\n",
    "            if not images == None:\n",
    "                for image in images:\n",
    "                    yield {\n",
    "                        'image': image.get('url'),\n",
    "                        'title': title\n",
    "                    }\n",
    "\n",
    "def save_image(item):\n",
    "    \"\"\"存储相关的图片\"\"\"\n",
    "    print(\"存储图片：\")\n",
    "    global count\n",
    "    if not os.path.exists(item.get('title')):\n",
    "        os.mkdir(item.get('title'))\n",
    "        # os.chdir(os.getcwd()+item.get('title'))\n",
    "    try:\n",
    "        response = requests.get(item.get('image'))\n",
    "        if response.status_code == 200:\n",
    "            filepath = item.get('title')+\"/\"+str(count)+\".jpg\"\n",
    "            if not os.path.exists(filepath):\n",
    "                with open(filepath,'wb') as f:\n",
    "                    f.write(response.content)\n",
    "                count+=1\n",
    "            else:\n",
    "                print('Already download')\n",
    "        # os.chdir(originalocation)\n",
    "    except requests.ConnectionError as e:\n",
    "        print(\"Error.\\n\"+str(e))\n",
    "from multiprocessing.pool import Pool\n",
    "import time\n",
    "def main(offset):\n",
    "    #print(\"mainfunction......\")\n",
    "    json = get_page(offset)\n",
    "    #print(json)\n",
    "    for i in get_image(json):\n",
    "        print(i)\n",
    "        save_image(i)\n",
    "\n",
    "START = 1\n",
    "END =20\n",
    "if __name__ =='__main__':\n",
    "    print('start......')\n",
    "    pol = Pool()\n",
    "    time.sleep(1)\n",
    "    groups =[x*20 for x in range(START,END+1)]\n",
    "    pol.map(main,groups)\n",
    "    pol.close()\n",
    "    pol.join()\n",
    "    print(\"over......\")\n"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n",
     "is_executing": true
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "注意：爬取的时候，因为访问过于频繁可能导致无法爬出数据，因此最好添加代理服务！\n",
    "\n"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "PyCharm (PycharmWorkspace)",
   "language": "python",
   "name": "pycharm-7b63a738"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}